Maximum segment size

The maximum segment size (MSS) is a parameter of the TCP protocol that specifies the largest amount of data, specified in octets, that a computer or communications device can receive in a single TCP segment, and therefore in a single IP datagram. It does not count the TCP header or the IP header.^[1] The IP datagram containing a TCP segment may be self-contained within a single packet, or it may be reconstructed from several fragmented pieces; either way, the MSS limit applies to the total amount of data contained within the final reconstructed TCP segment.

Therefore: Headers + MSS ≤ MTU

The Minimum MSS = Maximum datagram size - IP header size - TCP header size
So every IPv4 host is required to be able to handle an MSS of at least 536 octets (= 576 - 20 - 20)
and every IPv6 host is required to be able to handle an MSS of at least 1220 octets (= 1280 - 40 - 20).^[2]

For most computer users, the MSS option is established by operating system on the SYN packet during the TCP handshake. Each direction of data flow can use a different MSS.

Further reading

• Comer, Douglas E. (2006). Internetworking with TCP/IP. Vol. 1 (5/E ed.). Upper Saddle River, NJ, USA: Prentice Hall. 
• Kozierok, Charles M. (2005-09-20). The TCP/IP Guide (3.0 ed.). http://www.tcpipguide.com/free/t_TCPMaximumSegmentSizeMSSandRelationshiptoIPDatagra-2.htm. Retrieved 2011-09-08. 

References

1. ^ RFC 879, page 2, Section 3, "The MSS counts only data octets in the segment, it does not count the TCP header or the IP header."
2. ^ RFC 2460, page 28, Section 8.3